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Abstract 

The FACT telescope on the Canaries island of La Palma is the first Imaging Atmospheric Cherenkov Telescope (lACT) to use solid state photo¬ 
multipliers. It generates up to two terabytes of data per night which motivated us to investigate how to reduce the volume of data. Reducing the 
throughput enables us to efficiently acquire, store and process the observations data. This document presents the conclusions of this work, along with 
the implementation of the custom compression algorithm and I/O layer that is currently in use to operate the telescope. 

Keywords: Gamma Astronomy, Lossless Gompression, File Format. 


1. Introduction 

The First Geiger-mode Avalanche photodiode (G-APD) 
Gherenkov Telescope (FAGT) has been operating on the 
Ganary island of La Palma since October 2011. Opera¬ 
tions were automated so that the system can be operated 
remotel30 Manual interaction is required only when the 
observation schedule is modified due to weather conditions 
or in case of unexpected events such as a mechanical failure 
mm- Automatic operations enabled high data taking effi¬ 
ciency, which resulted in up to two terabytes of FITS files 
[3] being recorded nightly and transferred from La Palma 
to the FAGT archive at ISDG in Switzerland. Since long 
term storage of hundreds of terabytes of observations data 
is costly, data compression is mandatory. This paper dis¬ 
cusses the design choices that were made to increase the 
compression ratio and speed of writing of the data with 
respect to existing compression algorithms. 

Following a more detailed motivation, the FAGT com¬ 
pression algorithm along with the associated I/O layer is 
discussed. Eventually, the performances of the algorithm 
is compared to other approaches. 

2. Motivations 

A typical night of data taking generates up to two ter¬ 
abytes of raw data. Several compression algorithms were 
considered and tested to evaluate their performance on 
Gherenkov data. Lossy compression was discarded to re¬ 
tain all bits of information coming from the detector. As 


the first telescope using a new technology for photo detec¬ 
tion, this choice was important to avoid any bias in the 
characterization of the G-APDs. Insufficient throughput 
disqualified the slowest algorithms (Izma [4] and bzip2 [5]) 
even though they provided excellent compression, gzip [6] 
was selected for the commissioning of the telescope. 

The classical separation between raw file format and 
compression introduces several reads and writes to the 
storage which could be avoided if the file format would na¬ 
tively support compression. Several formats such as HDF5 
[ 7 ] and ROOT [8] support compression natively but are 
not widely employed by the astronomers community. On 
the other hand. Tile-compressed FITS [9] is a convention 
that allows to store compressed image data into binary ta¬ 
bles. A recent evolution of the convention enables to also 
compress binary tables natively m- This convention is 
fully FITS compliant with extra header keywords added 
to accommodate for the compression. 

The current Tile-compressed FITS implementation is 
available from the GFITSIO library and via a set of two 
executables, fpack and funpack, that can compress and 
decompress FITS file^ Various compression algorithms 
are already included m- Even though the compression 
of images was a fully functional feature, the handling of 
binary tables remained experimental at this time. More¬ 
over, the specific noise pattern of the analog ring buffer 
(see figure motivated us from investigating a specific 
way to compress this data. Our primary goal was to im¬ 
prove the compression ratio of raw data while maintaining 
a high throughput. We also wanted to explore the possi¬ 
bility to use calibration data of the analog ring buffer to 
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increase the compression ratio of raw data. To make sure 
that this calibration data is exploited in the best possible 
way it had to be applied by the I/O layer, thus making 
the compression process more complex. 


3. Compression 


An overview of the implemented I/O layer can be seen 
in figure The FACT algorithm works on sixteen-bits 
integer data and is lossless to allow to reconstruct the 
original raw data from the compressed output. Our al¬ 
gorithm consists of three separate steps. First the data is 
reversibly pre-calibrated to reduce the quasi random noise 
coming from the readout system to allow for high com¬ 
pression rates. Then the data is preconditioned such that 
number of bits needed for storage is reduced. Eventually, 
the information is transformed such that only these signif¬ 
icant bits are stored. These steps are discussed in more 


details in sections 3.1, 3.2 and 3.3 respectively. 



Fig. 1. Effect of DRS-calibration on a single pixel’s 
waveform samples. Top, red: raw data coming from the 
DRS4. Bottom: DRS calibrated values. Left: large, 60 
p.e. Cherenkov event. Right: small, 2 p.e. dark count 
event. The raw data outputted by the EACT camera is 
sampled by a 12-bits ADC and stored using 16-bits 
integers. The actual DRS calibration transforms the 
samples into floating-point values, while the values are 
truncated to integers for compression purposes. 



3.1. Drs Calibration 

The EACT Camera [T] employs a Domino Ring Sam¬ 
pler chip (DRS 4, [12]) to continuously record the signal 
from the detector plane. This chip is an analog ring buffer 
that stores the signal before it is digitized and acquired. 
Each DRS4 sample has an individual offset that must be 
calibrated using pedestal values. Upon trigger, the sam¬ 
pling is halted and the content of the buffer is readout. A 
trigger can happen at any time, thus the readout window is 
different from event to event. Therefore, not only the off¬ 
sets are individual to each sample but also vary from event 
to event. Although strictly speaking the offsets that are 
added to the raw signal are deterministic, they will appear 
as random to most algorithms and significantly decrease 
the achieved compression ratio. To compensate for this 
effect, the position of the readout window is recorded as 
well so that the raw data can be calibrated with the offsets 
individual to each sample before being analyzed [13]. In 
Eigurej^ (top) a raw signal is compared with a calibrated 
one (bottom) while the statistics for an entire event can 
be seen on figure [^ 

Applying this calibration reduces the amount of pseudo¬ 
random noise. The downside is that this moves the data 
from integer to floating-point space which would introduce 
additional bytes in each sample. To stay in integer space 
and allow to reverse the process, the compression algo¬ 
rithm only applies the integer part of the offset. Since 
the majority of offsets is larger than unity, this does not 
significantly change the width of the resulting distribution 
(by only 0.0011). Another advantage is that for a first 
look at the data this calibration is fairly sufficient and, 
therefore, allows easy access to semi-calibrated data. The 
offsets themselves are stored in each data file thus allow¬ 
ing to reverse this simplified calibration and apply a more 
precise one. This is entirely done on-the-fiy by the I/O 
layer in a transparent way for the user who is only aware 
of the raw values. 


Fig. 2. Overview of the FACT compression I/O layer. 
Each Cherenkov event undergoes three processings. First 
the data is calibrated (DRS). It is then preconditioned 
and eventually compressed using a Huffman encoder. 

3.2. Preconditioner 

While the DRS calibration step reduced the amount 
of noise, the data still contains a significant amount of 
pulses originating from so-called dark noise which require 
a comparably high number of bits for storage. Since these 
pulses look just like photons signal and thus have slow 
slopes compared to the sampling frequency, storing their 
derivative instead of their amplitude should reduce the re¬ 
quired number of bits. If just the difference between two 
consecutive samples would be recorded, the noise from two 
samples would sum up yielding increased random noise on 
the differences. The comparably slow change of the signal, 
however, allows us to average two consecutive samples and 
use them to calculate the difference to the following one. 
In this way the additionally introduced noise component 
is reduced by \/2. Averaging a higher number of samples 
does not significantly lower the additional noise, but be¬ 
comes sensitive to faster signals. Therefore, the average of 
two has been found to be the best compromise. 

The preconditioned samples pi are calculated from the 
DRS calibrated raw data samples Si by 

Si-1 + Si-2 
Pi=Si - - - 

with 

Po = So and pi = si 

In practice, this algorithm reduces the number of dif¬ 
ferent values, while the occurrence of values close to zero 
is increased (Figure]^ bottom). The width of the signal 
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is reduced from 65 to 35 and from 49 to 15 counts for raw 
and DRS-calibrated data respectively. 

The original waveform is restored by 


Si = Pi + 


Pi-1 +Pi-2 

2 


with 


Si = Pi and So = Po 


Raw Values Distribution Raw Values Histrograms 



Drs-Offsetted Values Distribution Drs-Offsetted Values Histrograms 



Fig. 3. Plotting of the pixel values from a single event. 
Each event has 1440 pixels and for each pixel 300 time 
samples of 0.5ns are recorded. On top are the values 
distribution and histograms for raw data, while below are 
the same plots for DRS-calibrated data. In orange are 
the input data while in blue are the output values of the 
preconditioning. The mean and std. dev. are (-1881.95, 
65.0274), (-0.25, 35.63), (5.53, 49.09) and (-0.01, 15.21) 
for raw, preconditioned raw, DRS-calibrated and 
preconditioned DRS-calibrated respectively. The second 
peak in the histogram of the raw data comes from 
artificially added time markers employed to synchronize 
the trigger patches. 


3.3. Huffman Encoding 

The last step of our compression algorithm consists of 
minimizing the number of bits needed to store the data. 
This is achieved via entropy coding and more specifically 
using Huffman coding m- The principle of this encod¬ 
ing is to associate codes to symbols, which lengths are 
inversely proportional to the symbols occurrence count. It 


has been widely used over the past 50 years, with well 
known applications such as JPEGj^and MPS0 

The length of the input symbols can vary depending on 
specific quantization parameters, especially for lossy com¬ 
pression. 8-bits words is the most commonly used symbol 
length for lossless compression. We chose 16-bits words as 
input to overcome the necessity to code the zeros in the 
majority of the most significant bytes of the 16-bit data. 

An alternative approach to overcome this shortcoming 
is the Rice algorithm m, which separates the high and 
low parts of the words before compressing them separately. 

4. File format 

As a file format, the FITS tile-compression conven¬ 
tion has been chosen. The use of Tile-compressed FITS 
[9] allowed us to reuse most of the analysis pipeline and 
FTOOLS. For instance, all data files are verified by the 
FTOOL fitsverify before being accepted to the long-term 
archive. Our own FITS I/O layer had already been devel- 
opped within the FACT project for performances reasons 
and more particularily to be able to control the memory al¬ 
locations. These classes were extended to accomodate for 
the compression algorithms and Tile-compression conven¬ 
tion. To implement the compressed format it was enough 
to derive from the existing classes, thus reducing the re¬ 
quired development work to a minimum. 

FITS-files are organized in extensions. Each extension 
starts with an ASCII header that defines the structure and 
length of the data stored in the current extension. Headers 
are directly followed by fixed-length data organized into 
columns and eventually comes the variable-length data in 
a section called the heap. The heap contains compressed 
data that could not be stored in the fixed-length columns 
of the main data table. In the compressed format, rows are 
compressed in groups of n (called a tile) and each column 
from a group of rows is compressed separately, as directed 
by the Tile-compressed FITS convention. 

A schematic view of the file format can be seen in fig¬ 
ure In Tile-compressed FITS files, all compressed data 
goes into the heap as it is naturally variable in length. In 
our implementation we also allow for the storage of fixed- 
length data in the heap. This allows to read continuous 
sectors on the disk to retrieve a full set of rows, rather 
than alternating reads between the columns and heap ar¬ 
eas. In this way the streaming capability of the format 
is kept. Moreover, storing all data to the heap simplifies 
the structure of the I/O code, thus easing the long-term 
maintenance. 

A few minor adaptation were implemented to address 
some shortcomings of the Tile-compressed FITS format 


^http: //www.itu.int/rec/T-REC-T.81-199209-I/en 
^http: / / mpeg.chiariglione.org/standards/mpeg-1 / audio 
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Fig. 4. Schematic view of the data layout on disk. The 
file format complies with the Tile-compressed FITS 
convention. All columns, even uncompressed ones are 
moved to the heap area for better performances. The 
main data table thus becomes a catalog. The tile and 
block headers allow to reconstruct the file’s catalog from 
the heap only. This is used in emergency situations 
where not enough space was reserved. It could also be 
useful if each block of events (tiles) are stored in 
relational databases (RDMS). This way complete FITS 
files could be efficiently reconstructed on the fiy based on 
the request made to the RDMS. 


when working with streams rather than data sets. These 
change do not modify the capabilities of FITS but rather 
are meant to make operations more robust and less re¬ 
sources intensive. They are discussed in details in the fol¬ 
lowing sections. 

4 . 1 . Custom FITS keywords 

Two new header keywords were introduced: RAWSUM 
and ZSHRINK. 

RAWSUM. In the standard implementation of the Tile- 
compressed FITS format, the checksum of the uncom¬ 
pressed table (DATASUM) is saved during compression 
by the fpack tool. 

While in the FITS convention this checksum is cal¬ 
culated from big-endian data, the data arriving from the 
telescope is little-endian. To avoid the need of an addi¬ 
tional byte swapping just to calculate the DATASUM, a 
new keyword RAWSUM has been introduced storing the 
checksum of the uncompressed, little-endian data. Appart 
from the omission of the byte-swap, the computation of 
both DATASUM and RAWSUM is identical. This new 
keyword does not forbid to use the usual CHECKSUM 
keywork to verify the integrity of the compressed data, 
which we do. 

ZSHRINK. In our Tile-compressed FITS streamer the main 
data table is used to store pointers to the heap-area, usu¬ 
ally one for each compressed row. Since this table is stored 


before the heap-area, it implies that the number of rows 
to be written to the file is known in advance. This is not 
always possible because experiments often prefer to group 
data per interval of time rather than per unit of raw data. 

As a remedy, an estimate of the expected number of 
rows/events is calculated when the file is opened, and a 
corresponding number of bytes reserved. If the number of 
events grows larger than the number of pointers which can 
be written to the table, only every A^th pointer is written 
to the table. The integer N is then stored in the keyword 
ZSHRINK. 

To still be able to read the compressed data entirely, 
specific markers are put in the heap area that allow us to 
not only read the compressed data without access to the 
main data table, but also to reconstruct this table entirely 
from the heap area only. 

4 . 2 . Compression bloeks 

Inside the heap, blocks of compressed data start with 
their own variable-length header which is described in fig¬ 
ure An ordering field allows us to order the data either 
by row or by column. In the case of FACT, ordering the 
data per row yields a better compression ratio and less 
data shuffling. 


byte byte byte 

0 8 10 

I I I I I I I I I I I I I r""' 

'-;-^1 I'-;- 

size ordering I processings 
procs 

Fig. 5. Compressed block header, size (8 bytes) is the 
size in bytes of the block, ordering (1 byte) is how the 
data was copied from the columns. ASCII code for ’R’ 
means that the data was copied as is, while ’C’ means 
the it was transposed as directed by the Tile-compressed 
FITS convention, numprocs (1 byte) is the number of 
compression algorithms that were applied and 
processings {numprocs bytes) is the identifiers of the 
applied algorithms. Currently the only valid values are 
0 = rare, 1 = smoothing and 2 = Huffmanl6. 

Besides block headers, tile headers are interleaved with 
the compressed data. If the main data table is complete, 
i.e. contains pointers to all rows, these headers are redun¬ 
dant. If more rows are written compared to the space re¬ 
served initially (ZSHIRNK> 1), they allow to reconstruct 
the missing main data table entries. They can also be used 
to recover the catalog upon data corruption. 

4.3. DRS-calibration 

The DRS-calibration is not applied by the compres¬ 
sion algorithm per se. The calibration offsets are stored 
in a separate Tile-compressed table in the same FITS file 
as the data itself. This introduces an overhead which is 
quickly absorbed by the increased compression ratio. For 
optimal efficiency, the calibration table is always placed 
before the main data table. This allows the data access 
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byte byte byte byte 



Fig. 6. Tile header, tilemarker is an identifier marking 
the start of a new tile. Its content is the ASCII codes of 
TILE, numberofrows is the number of rows compressed 
in the current tile, size is the total size of the tile in 
bytes. 

layer to quickly find the calibration table without having 
to skip through the file. The data access layer finds the 
calibration table by looking for a table called ZDrsCellOff- 
sets^ and thus this name should not be used for another 
table. 

4.4- Access Layer 

The access layer was written in C++ and consists of 
classes that inherit from each other, as follow: factfits 
zfits fits. Additionally, fits has the zlib as a link op¬ 
tion which enables it to read gzipped FITS natively, zfits 
can read everything fits can plus Tile-compressed FITS 
and factsfits can read all of what zfits can plus DRS- 
calibrated FITS. The source code of the classes can be 
checked-out from the FACT svrj^ All reading classes are 
single-threaded while their writing counterparts use mul¬ 
tiple cores for fast compression. 

The writing is done as follow: incoming rows are buffered 
until the target number of rows per tiles is reached. Then 
the buffer is given to a compression thread while a new 
one is allocated to receive new events. Each compression 
thread will shuffle and compress the data before passing 
the compressed rows to a thread that perform the actual 
write to disk. 

Reading occurs as follow: compressed tiles are loaded 
to memory, decompressed and the raw data is buffered 
until it is requested by the users or until another tile is 
read. 

Our I/O layer allows to start reading a file before it 
has been closed by the writing process. This proved to 
be useful to start real-time analysis as soon as possible 
and to provide statistics to the operator if they need to be 
extracted from raw data files. 

5. Results 

The FACT compression algorithm was compared to 
other de facto standards in the astrophysics community. 
Since the DRS calibration is specific to the DRS 4 readout, 
two separate input data sets were employed: raw events 
and their DRS-calibrated version. The calibration step 
was pre-calculated for all tests so that the comparison is 
fair. The tested file formats were: 


^https://www.fact-project.org/svn/trunk/FACT++ 


• EVENTIO is the format used to produce Monte- 
Carlo simulations for the CTA projecf|^ This for¬ 
mat tightly packages the input data using a internal 
compression algorithm. 

• FITS the reference file format where the data is writ¬ 
ten in plain binary. Only the bytes order might be 
modified to make the data big-endian. 

• Tile-compressed FITS the FITS convention that sup¬ 
ports binary tables compression natively. 

In addition, HDF5 and ROOT files were considered 
but not tested because the way they package the data over- 
lapps with the above formats. HDF5 defines the file format 
but not the compression algorithm to be used per-se, while 
ROOT uses a derivative of gzip. FITS was used as a data 
source format, while Tile-compressed FITS was split into 
two flavors: 

• Rice: the FTOOLS fpack and funpack were used to 
(un)compress the data using the Rice compression 
algorithm m- This algorithm splits high and low 
bytes of the input values before compressing them 
separately. 

• Fact: the compression algorithm and I/O layer de¬ 
scribed in this paper were used to produce Tile- 
compressed FITS. 

Besides file formats, each produced file has been further 
compressed using several classical algorithms, namely: 

• Izma the Lempel-Ziv-Markov chain-Algorithm that 
was under development until 2001. 

• gzip the well known algorithm widely used by the 
linux community. It employes the LZ77 and Huffman 
coding 

• hzip2 the more recent algorithm meant as an alterna¬ 
tive to gzip. It uses the Burrows-Wheeler transform 
and Huffman coding. 

All programs that have compression level options were 
set to their minimum. Preliminary tests conducted with 
higher levels showed that the achieved ratios did not in¬ 
crease significantly whereas the processing time did. 

Tests were conducted on data from 2 nights: 2014/01/01 
and 2014/01/10. The first night was dark, while the sec¬ 
ond had moonlight for half of the night resulting in more 
recorded background photons and thus a higher noise level. 
All I/O operations were done to/from the shared memory 
to alleviate caching effects and storage bottlenecks. Only 
files small enough to fit in the available memory were pro- 
cessecQ Since only the FACT I/O layer could handle the 


®http://www.mpi-hd.mpg.de/hfm/ bernlohr/iact-atmo/ 

^71 out of 261 runs could not fit in the available 64GBytes of 
memory 
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DRS-calibration natively, the input data sets were pre- 
processed before the other compression algorithms were 
applied. The size of the DRS-calibration table was ig¬ 
nored in the calculations. This table has a constant size of 
approximately 1.4MB compressed or 0.02 percent of the 
average raw data file, to be added to the output size of 
the compressed files. The raw throughput of the memory 
was approx. 1.5GB/s thus the performances given below 
reflect the computation time rather than memory 10. 



FITS 

EventIO 

Rice 

FACT 

native 

n/a 

200.51 

109.29 

120.34 

Izma 

30.75 

34.16 

31.86 

32.13 

gzip 

94.14 

86.64 

83.92 

90.57 

bzip2 

26.80 

28.51 

34.05 

34.77 


Table 1 

Mean decompression throughput in MB/s of input DRS-calibrated 
data. 


Ratio vs Throughput - raw input 



MB/S 


♦ Izma 

♦ gzip 

♦ bzip2 
Aevtio 

A evtio+Izma 
A evtio+gzip 
A evtio+bzip2 

♦ rice 

♦ rice+Izma 

♦ rice+gzip 

♦ rice+bzip2 
Xfact 

X fact+Izma 
X fact+gzip 
X fact+bzip2 


Fig. 7. Obtained compression ratio compared to 
throughput for raw data set. The best performances are 
obtained by the data points with the largest x and y 
values. 


♦ Izma 

♦ gzip 

♦ bzip2 
A evtio 
A evtio+Izma 
A evtio+gzip 
A evtio+bzip2 

♦ rice 

♦ rice+Izma 

♦ rice+gzip 

♦ rice+bzip2 
Xfact 

X fact+Izma 
X fact+gzip 
X fact+bzip2 

Fig. 8. Obtained compression ratio compared to 
throughput for DRS-calibrated data sets. 

A summary of the compression performances can be 
seen on figures and The compression ratio for the 
raw data went up to 2.25, while the I/O layer delivers the 
best average ratio and an average throughput of 131MB/s. 
For Drs-calibrated data, we achieved ratios up to 3.5 with 
Rice. The FACT algorithm is a close contender, achiev¬ 
ing an average ratio of 3.39 coupled with a throughput of 
143MB/S. 

Decompression speed for calibrated data can be seen 
on table [TJ EventIO turned out to be the fastest format 
when it comes to reading the data back, topping out at 
200 MB/s. The FACT algorithm was second at 132 MB/s 
and Rice arrived third at 115 MB/s. Classical algorithms 
performed much worse with only gzip coming close to the 


Comp. Ratio vs Throughput - DRS-calibrated ii 



MB/s 


Rice decompression performances. Detailed performances, 
including decompression of raw data, are given in the an¬ 
nex 


6. Discussion 

The tests have shown that the described compression 
algorithm provides good performances when applied to 
the data produced by the FACT telescope. Compared to 
the previously used gzipped-FITS format, it allowed the 
experiment to improve the compression ratio of its raw 
data from 1.6 to 3.4 while the compression throughput 
went from 33.6 MB/s up to 143.0 MB/s. Only Rice was 
able to outperform our algorithm under specific conditions, 
namely for DRS-calibrated data when the lighting condi¬ 
tions are not optimal. These degraded lighting conditions 
occur between the nautical and astronomical twilight, and 
when the moon is up. This suggests that the Rice al¬ 
gorithm is very good at compressing actual signal while 
our algorithm achieves larger compression ratios with raw 
data. 

Adding classical compression on top of the custom ones 
did not significantly improve the results and even decreased 
the compression ratio in some cases. This was expected as 
the first compression stage leave mostly noise in the data 
set. 

The FACT algorithm is the fastet of the tested ap¬ 
proaches, while Rice is second. The classical algorithms 
were much slower and would be difficult to use for real¬ 
time operations as they would require that the data is 
compressed after data taking in a separate step or using 
a much larger number of compute cores. The throughput 
obtained by these algorithms is faster in some cases where 
the data was first transcoded to a natively compressed for¬ 
mat. 

The best overall compression ratios were obtained by 
combining a native algorithm - either Rice or Fact - and 
gzip. However, the gain in compression ratio of about 1% 
is not significant enough to accept a decrease in processing 
speed of more than 50%. 

Considering the good performances of the Rice algo¬ 
rithm under poor lighting conditions for DRS-calibrated 
data, an additional gain could be achieved. However, given 
the marginal improvements compared to our algorithm, 
that only a small fraction of the data is taken under these 
conditions and the additional complexity to apply two sep- 
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arate compressions, the FACT algorithm remains a good 
choice. 

7. Conclusion and Future Work 

In this paper a simple compression algorithm was pre¬ 
sented which provides good performances when applied to 
FACT data. This algorithm was implemented in C++ and 
integrated into a streaming I/O layer that produces Tile- 
compressed FITS file format, which makes it suitable for 
the real time operations of lACTs. 

The performances of our algorithm was compared with 
existing approaches. The experience gained during this 
work will be reused while devising the raw data format 
for the Cherenkov Telescope Array to ensure that the best 
compression ratios achievable in real-time is implemented. 

The I/O layer described in this paper has been used 
since more than three years for the datataking of FACT. 
The total amount of encoded data is currently more than 
600TB uncompressed and all raw data can be read without 
any problem. 
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Appendices 

A. Detailed results 

The detailed results are presented below. Eirst the 
compression ratios and then the compression throughput. 
One plot with the tested formats is presented per classical 
compression ratio. In the case of the raw file format, plain 
EITS was omitted as the corresponding compression ratio 
is always one and calculating a throughput would make no 
sense. 

Eor both the compression ratios and throughput, the 
data points are organized per run. Each run corresponds 
to a single raw data file that was moved to shared memory 
in plain EITS. All the code was compiled with gee 4.4.7 
on scientific linux 6.2 x64. The optimizer was set to -02. 
Tests were made with -03, but the lack of performances 
improvements made us stay with the -02 option. 

The first half of each plot correspond to the night of 
2014/01/01 (dark) up to run 78 while the second half cor¬ 
responds to the night of 2014/01/10 (moon). Jumps in the 
compression ratios correspond to repointings of the tele¬ 
scope, while jumps in the throughput are most likely due 
to system interrupts of the operating system of the server 
onto which the tests were run. Indeed, despite the fact 
that we made sure that no other processing was running 
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FITS 

EventIO 

Rice 

FACT 

native 

1 

1.75 

2.11 

2.25 

Izma 

1.86 

2.07 

2.09 

2.25 

gzip 

1.57 

2.01 

2.11 

2.27 

bzip2 

2.10 

2.16 

2.10 

2.25 


Table 2 

Mean compression ratio for raw input data. 



FITS 

EventIO 

Rice 

FACT 

native 

1 

1.97 

3.48 

3.39 

Izma 

2.53 

2.27 

3.49 

3.44 

gzip 

2.11 

3.14 

3.51 

3.45 

bzip2 

3.26 

3.42 

3.48 

3.42 



FITS 

EventIO 

Rice 

FACT 

native 

n/a 

95.97 

129.33 

142.96 

Izma 

7.67 

9.47 

16.89 

16.94 

gzip 

47.11 

41.21 

50.05 

52.52 

bzip2 

10.36 

14.70 

16.89 

16.59 


Table 5 

Mean compression throughput in MB/s of input DRS-calibrated 
data. 


Table 3 

Mean compression ratio for DRS-calibrated data. 

on the servers for the tests, some system interrupts still 
occurred. The total input size was 1330.56GB while the 
average file size was 6.97GB. 

Tables and show the average compression ratios for 
the raw and DRS-calibrated input data sets respectively. 
A summary of the compression speeds can be seen on table 
and Eventually, the decompression speed can be seen 
on tables [6] and [T] 



FITS 

EventIO 

Rice 

FACT 

native 

n/a 

193.63 

114.87 

131.90 

Izma 

21.10 

22.51 

21.66 

23.65 

gzip 

79.55 

82.58 

85.75 

95.53 

bzip2 

23.72 

24.10 

23.99 

26.07 


Table 6 

Mean decompression throughput in MB/s of input raw data. 


A.l. Compression Ratios 

Figures [9| and show the compression ratios obtains 
for the raw data set. They correspond to the transcoding 
to the native file format, Izma, gzip and bzip2 versions 
respectively. Figures \TT\ and 12 follow the same ordering, 
only for the DRS-calibrated input data set. 


A.2. Compression Speed 

Figuresandshow the compression speed in MB/s 
obtains for the raw data set. They correspond to the 
transcoding to the native file format, Izma, gzip and bzip2 
versions respectively. 

Figures and follow the same ordering, only for 
the DRS-calibrated input data set. 



FITS 

EventIO 

Rice 

FACT 

native 

n/a 

95.68 

128.84 

131.05 

Izma 

6.68 

8.80 

10.75 

11.52 

gzip 

33.62 

28.11 

39.27 

41.47 

bzip2 

8.62 

11.01 

10.61 

11.02 


Comp. Ratio - Raw - native 



Comp. Ratio - Raw - Izma 



Run Number 


Fig. 9. Gompression ratios of raw data runs for native 
and Izma output respectively. 


Table 4 

Mean compression throughput in MB/s of input raw data. 
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Comp. Ratio - Raw - gzip 


Comp. Ratio - DRS Offset - native 




Comp. Ratio - Raw - bzip2 


Comp. Ratio - DRS Offset - Izma 




Fig. 10. Compression ratios of raw data runs for gzip Fig. 11. Compression ratios of DRS-calibrated data 

and bzip2 output respectively. runs for native file format and Izma output respectively. 
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Comp. Ratio - DRS Offset - gzip 


Throughput - Raw - native 




Comp. Ratio - DRS Offset - bzip2 Throughput - Raw - Izma 




Fig. 12. Compression ratios of DRS-calibrated data Fig. 13. Compression throughput in MB/s of raw data 

runs for gzip and bzip2 output respectively. runs for native and Izma output respectively. 
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MB/S MB/S 


Throughput - Raw - gzip 



Run Number 


Throughput - Raw - bzip2 



Throughput - DRS Offset - native 



Run Number 


Throughput - DRS Offset - Izma 



Fig. 14. Compression throughput in MB/s of raw data 
runs for gzip and bzip2 output respectively. 


Fig. 15. Compression throughput in MB/s of 
DRS-calibrated data runs for native and Izma output 
respectively. 
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Throughput - DRS Offset - gzip 



Run Number 


Throughput - DRS Offset - bzip2 



Fig. 16. Compression throughput in MB/s of 
DRS-calibrated data runs for gzip and bzip2 output 
respectively. 
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